Automatic Robustness Testing of Off-the-Shelf Software Components

Mission-critical system designers are turning towards Commercial Off-The-Shelf (COTS) software to reduce costs and shorten development time even though COTS software components may not specifically be designed for robust operation. (Systems are robust if they can function correctly despite exceptional inputs or stressful conditions.) Automated testing can assess component robustness without sacrificing the cost and time advantages of using COTS software. This report describes a scalable, portable, automated robustness testing tool for component interfaces. An object-oriented approach based on parameter data types rather than component functionality essentially eliminates the need for functionspecific test scaffolding. A full-scale implementation that automatically tests the robustness of 233 operating system software components has been ported to nine POSIX systems. Between 42% and 63% of components on the POSIX systems measured had robustness problems, with a normalized failure rate ranging from 10% to 21% of tests conducted. Robustness testing could be used by developers to measure and improve robustness, or by consumers to compare the robustness of competing COTS component libraries. Acknowledgments This research was supported under DARPA contract DABT63-96-C-0064 (the Ballista project) and ONR contract N00014-96-1-0202. Thanks to John DeVale and Jiantao Pan for their help in collecting experimental data.